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INTRODUCTION 

In  this  project,  we  are  investigating  a  stereoscopic  mammography  system  we  have 
developed  using  state-of-the-art  digital  technology  that  enables  a  mammographer  to  view  the 
internal  structure  of  the  breast  directly  in  depth.  Stereo  mammograms  are  obtained  by  taking  two 
x-rays  of  the  breast  from  viewpoints  separated  by  about  6  degrees,  each  view  captured  by  a  new 
GE  digital  full  field-of-view  mammography  unit.  The  two  digital  images  are  viewed  on  a  stereo 
display  workstation  by  a  radiologist  wearing  stereo-viewing  eye-glasses.  The  system  provides 
the  user  with  control  over  many  display  parameters  such  as  brightness,  contrast,  grayscale 
polarity,  and  inversion  of  depth.  The  primary  aims  of  this  project  are  to  further  develop  and 
refine  this  system  and  then  to  evaluate  its  effectiveness  in  increasing  the  accuracy  of  diagnosis  of 
breast  cancer  relative  to  the  diagnostic  accuracy  afforded  by  standard,  non-stereo  film  views. 

The  expected  increase  in  diagnostic  accuracy  could  increase  the  yield  of  biopsy  substantially  by 
allowing  the  radiologist  to  more  confidently  recommend  accelerated  follow-up  of  truly  benign 
disease  and  biopsy  of  truly  malignant  disease. 

BODY  OF  REPORT 

In  Year  3  of  this  project,  we  have  made  progress  in  six  major  areas:  (1)  continued  accrual 
of  stereo  mammographic  images  from  patients  enrolled  in  this  project,  (2)  development  of 
computer-assisted  methods  for  processing  of  the  raw  stereo  images,  (3)  improvements  to  the  user 
interface  of  the  stereo  display  workstation,  (4)  further  research  on  the  design  of  a  new  stereo 
display  system,  (5)  identification  and  development  of  stereo-based  visual  features,  and  (6)  design, 
conduction  and  analysis  of  a  preliminary  reading  study.  We  discuss  our  work  in  each  of  these 
areas  below. 

1.  Accrual  of  Stereo  Mammographic  Images  from  Patients  Enrolled  in  this  Project  (Task  4) 

To  date,  we  have  obtained  stereo  mammographic  images  of  166  lesions  from  patients 
enrolled  in  this  project.  The  breakdown  of  stereo-imaged  lesions  is  shown  below  in  Table  1  for 
152  of  the  166  imaged  lesions.  (Of  the  other  14  lesions,  we  have  not  yet  received  the  pathology 
reports  on  3,  and  11  cases  have  been  dropped  either  because  the  diagnostic  films  are  not  available 
to  us  or  because  the  lesion  was  palpated  but  not  seen  mammographically.)  Of  the  152  lesions 
with  known  pathology,  48  are  malignant  and  104  are  benign.  With  respect  to  mammographic 
appearance  of  the  lesion,  63  of  the  lesions  are  seen  as  a  mass  (with  or  without  associated 
calcifications,  77  as  microcalcifications  alone,  and  12  are  seen  as  architectural  distortion  (with  or 
without  associated  microcalcifications). 


Table  1.  Distribution  of  imaged  lesions  by  type  and  pathology. 


Lesion  Type 

Benign 

Pathology 

Malignant 

Total 

Mass 

32 

22 

54 

Mass  +  calcifications 

3 

6 

9 

Calcifications 

65 

12 

77 

Architectural  Distortion 

3 

5 

8 

Arch.  Dist.  +  calcifications 

1 

3 

4 

Total 

104 

48 

152 
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In  addition  to  the  pair  of  digital  stereo  images  obtained  for  each  enrolled  patient,  we  are 
continuing  to  collect  three  documents:  (1)  a  study  form  filled  out  by  the  mammographer 
indicating  the  radiographic  nature  of  the  biopsied  lesion  and  its  locus  in  the  cranio-caudal  view 
(the  point-of-view  of  the  stereo  mammogram),  (2)  the  full  mammographic  report  from  the  study 
that  led  to  the  biopsy,  and  (3)  the  final  pathology  report. 

We  have  completed  the  development  of  a  case  database  for  the  project.  The  database 
contains  fields  of  relevant  information  for  each  case,  including  serially-assigned  case  study 
number,  date  of  diagnostic  study,  date  of  stereo  imaging/biopsy,  breast  biopsied,  mammographic 
lesion  type,  lesion  locus,  pathology  truth,  and  several  parameters  describing  the  transformation  of 
the  raw  digital  images  to  a  viewable  stereo  pair.  We  will  continue  to  populate  this  database  with 
new  cases  as  they  are  acquired. 

2.  Development  of  Computer-Assisted  Methods  for  Processing  Raw  Stereo  Images  (Task  4) 

During  Year  3,  we  developed  a  software  application  to  semi-automate  the  processing  of 
raw  stereo  images  into  viewable  stereo  pairs.  Each  digital  mammographic  image  captured  on  the 
GE  unit  at  UMMC  is  2304  pixels  wide  by  1800  pixels  high,  with  a  grayscale  resolution  of  16  bits 
stored  in  a  16-bit  word.  Captured  digital  images  are  stored  as  16-bit  TIFF-format  files  and  burned 
onto  a  CD-ROM  which  is  then  sent  to  BBN.  Because  of  limitations  of  the  stereo  display  card  in 
our  workstation,  we  need  to  process  the  raw  digital  images  to  reduce  the  size  of  each  image  to 
1024  by  1024  pixels  and  to  reduce  the  grayscale  from  16  bits  to  8  bits. 

A  number  of  steps  are  involved  in  this  processing.  First,  the  left-eye  image  of  a  pair  is 
opened  and  visually  examined  to  judge  the  adequacy  of  the  image.  Then,  an  image-processing 
algorithm  is  applied  to  the  image  to  segment  the  breast  tissue  from  the  surround.  The  user  can 
iteratively  modify  the  threshold  used  in  this  process  until  a  satisfactory  segmentation  is  achieved. 
The  program  then  calculates  a  cumulative  histogram  of  grayscale  values  for  the  pixels 
representing  breast  tissue  and  determines  the  minimum,  median,  and  maximum  grayscale  values. 
This  process  is  then  repeated  for  the  right-eye  image.  The  program  then  determines  the  offset 
between  the  median  grayscale  value  of  breast  tissue  in  the  two  images  and  applies  it  as  an 
additive  correction  to  all  of  the  pixels  in  the  appropriate  one  of  the  two  images,  in  order  to 
equalize  their  median  values.  We  found  this  correction  necessary  because  software  within  the 
GE  mammographic  unit  apparently  attempts  to  optimize  mapping  of  x-ray  density  to  grayscale 
usage  independently  for  each  newly  acquired  mammogram.  Consequently,  we  frequently  find 
differences  in  the  grayscale  histograms  between  the  two  images  of  a  stereo  pair,  even  though  the 
two  images  are  of  the  same  breast,  with  no  breast  re-compression  between  images.  The  only 
difference  in  the  two  images  is  the  small  6-degree  movement  of  the  x-ray  source  between  images. 
We  have  also  encountered  systematic  changes  in  the  use  of  the  16-bit  grayscale  range  over  the 
past  months  of  image  acquisition.  We  believe  that  these  are  due  to  frequent  updates  of  the 
image-capture  software  in  the  GE  mammographic  unit  as  the  GE  engineers  leam  more  about  the 
performance  of  the  unit. 

In  the  next  step,  the  program  reduces  the  grayscale  of  both  images  from  16  bits  to  8  bits. 

A  linear  re-mapping  of  the  grayscale  is  used,  extending  from  the  minimum  16-bit  grayscale  value 
to  the  maximum  16-bit  value.  The  transformed  versions  of  both  images  are  saved  as  8-bit  TIFF 
files.  The  program  next  saves  a  stereo- viewable  1024  x  900  pixel  overview  of  each  of  the  two 
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images  by  sampling  every  other  pixel  and  every  other  line.  This  pair  of  images  is  useful  to  the 
mammographer  to  obtain  a  stereo  view  of  the  entire  breast,  albeit  at  reduced  spatial  resolution. 

The  program  then  places  a  1024  by  1024  region-of-interest  outline  into  the  full  left-eye 
image  and  permits  the  user  to  move  this  outline  around  within  the  larger  image  until  the  user  is 
satisfied  that  the  lesion  and  surrounding  tissue  are  appropriately  enclosed  within  the  window. 

The  coordinates  of  the  user-selected  location  of  the  outline  are  recorded,  and  the  region-of- 
interest  is  then  extracted  and  saved  as  a  1024  x  1024  8-bit  TIFF  file.  This  process  is  repeated  for 
the  right-eye  image,  except  that  the  coordinates  used  for  positioning  the  1024  x  1024  window  in 
the  image  are  the  same  ones  determined  from  the  left-eye  image.  This  pair  of  images  is  of 
primary  importance  to  the  mammographer  since  it  permits  a  stereo  view  of  the  lesion  at  full 
spatial  resolution. 

3.  Improvements  to  the  User  Interface  of  the  Stereo  Display  Workstation  (Task  3) 

In  Year  3,  we  completed  the  design  of  the  user  interface  for  the  stereo  display  workstation 
and  rewrote  the  software  application  that  implements  this  design.  We  have  included  the  User’s 
Manual  for  the  application  as  Appendix  A.  During  the  year,  the  design  was  refined  based  on 
comments  from  our  expert  mammographers  as  they  used  the  system.  The  design  allows  the  user 
of  the  display  workstation  to  control  it  using  either  a  3-button  mouse,  in  combination  with  a  few 
marked  keys  on  the  system  keyboard,  or  by  point-and-click  interactions  with  controls  displayed 
on  the  system  monitor  (not  the  image  display  monitor).  Most  operations  can  be  executed  in  two 
ways:  either  using  the  mouse  by  itself  (Mouse  Mode)  or  by  clicking  on  displayed  controls  on  the 
system  monitor  (GUI  Mode).  A  guiding  principal  in  the  design  of  the  interface  has  been  to  make 
the  most  frequently  used  operations  immediately  and  conveniently  accessible  to  the  user,  and  to 
move  infrequently  used  operations  off  the  main  window  onto  secondary  windows  accessible 
from  a  pull-down  menu. 

We  have  removed  speech-control  of  the  system  from  the  application.  Our  experience 
with  using  speech  during  the  first  two  years  of  the  project  revealed  that  mammographers  did  not 
like  having  to  wear  a  headset  with  attached  microphone  and  they  did  not  like  having  to  use  a 
push-to-talk  button  to  enable  the  speech-recognition  system.  The  possible  gains  from  hands-free 
spoken  control  over  some  parts  of  the  system  were  more  than  offset  by  the  inconveniences  and  by 
imperfect  speech  recognition. 

An  important  conclusion  we  reached  in  our  human  factors  studies,  conducted  during 
Years  2  and  3,  was  that  control  over  continuous  parameters,  such  as  brightness  and  contrast,  is 
best  accomplished  using  movements  of  a  mouse.  In  our  system,  horizontal  movement  of  the 
mouse  controls  displayed  brightness  while  vertical  movement  controls  displayed  contrast.  We 
have  found  that  users  quickly  develop  a  cognitive  model  Of  this  2-dimensional  space  in  which  the 
mouse  is  placed,  and  can  move  it  effectively  to  achieve  a  desired  combination  of  brightness  and 
contrast.  As  a  further  aide,  when  in  this  mode,  the  location  of  the  mouse  in  the  2-dimensional 
space  is  shown  by  a  cursor  on  the  system  display  screen.  An  optimal  brightness/contrast  setting, 
saved  individually  for  each  case,  is  indicated  by  an  icon  located  at  the  appropriate  position  on  the 
system  display.  This  provides  the  user  with  a  landmark  to  move  towards  in  case  they  lose  track 
of  where  they  are  in  the  control  space.  In  addition  to  controlling  brightness  and  contrast,  the  user 
can  invert  the  grayscale  by  pressing  the  left  mouse  button,  and  invert  depth  by  pressing  the  right 
button. 
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A  number  of  other  controls  have  been  made  available  to  the  user.  In  stereo  display  mode, 
the  system  is  able  to  display  only  512  of  the  1024  available  lines  of  the  stereo  image.  In  order  to 
see  other  parts  of  the  image,  the  user  can  hold  down  the  up  or  down  arrow  keys  to  scroll  the 
image.  The  user  can  also  choose  to  zoom  the  image  by  factors  of  IX,  2X  or  4X.  The  user  can 
enable  a  crosshair  cursor  that  can  be  positioned  in  a  frontal  plane  by  movements  of  the  mouse. 
The  location  of  that  plane  in  depth  can  be  moved  back  and  forth  by  holding  down  the  left  or  right 
mouse  buttons.  This  in-depth  cursor  is  very  useful  with  a  group  of  users  viewing  the  display 
when  one  user  wishes  to  point  to  particular  structures.  Using  the  middle  mouse  button,  the  user 
can  also  draw  line  segments  or  freehand  lines,  in  depth,  to  point  to  or  outline  areas  of  interest 
within  the  display  volume. 

4.  Further  Research  on  the  Design  of  a  New  Stereo  Display  Workstation  (new  task) 

In  Year  3,  we  have  continued  the  task  of  designing  a  new  stereo  display  workstation.  The 
current  system  is  based  on  a  Matrox  Image-1280  display  card  that  is  no  longer  manufactured  by 
Matrox,  and  requires  an  EISA-bus  PC  that  is  now  obsolete.  More  recent  image  display  cards 
introduced  by  Matrox  do  not  directly  support  stereo  display.  However,  several  other  companies 
manufacturing  high-end  display  cards  have  begun  including  direct  support  for  stereo  displays  in 
one  or  more  of  their  PC  display  controller  cards.  Last  year,  we  identified  several  display 
controller  cards  that  include  support  for  stereo  display:  (1)  Gloria-XXL  (ELSA  AG),  (2)  Oxygen 
3D  (Dynamic  Pictures,  Inc.),  and  (3)  Fire  GL  Pro  (Diamond).  This  year,  we  have  identified 
another  card  that  will  soon  be  released,  the  P1760S  card  from  Metheus/Barco.  This  controller 
seems  the  most  promising  since  it  was  designed  with  medical  imagery  in  mind. 

We  have  also  been  investigating  a  new  monitor  to  replace  the  current  Hitachi  21”  color 
monitor.  In  the  course  of  conducting  a  Preliminary  Reading  Study  this  year,  described  below,  we 
have  found  that  the  spatial  resolution  of  our  displayed  stereo  images  is  being  degraded  by  the 
shadow  mask  of  the  color  monitor.  The  solution  to  this  problem  is  to  replace  the  color  monitor 
with  a  high-resolution  grayscale  (monochrome)  monitor.  We  have  determined  that  three 
companies  are  making  such  monitors,  mostly  for  the  medical  imaging  market:  (1)  Barco,  (2) 
Siemens,  and  (3)  Clinton  (Orwin).  We  are  currently  exploring  whether  any  of  these  monitors  are 
able  to  meet  the  requirements  for  stereo  imaging;  namely,  ability  to  synch  to  a  120  Hz  vertical 
refresh  rate,  capability  to  adjust  vertical  picture  height  to  one-half  screen,  and  incorporating  a 
fast-decaying  phosphor  to  prevent  image  ghosting  during  stereo  viewing. 

The  goal  is  to  configure  a  new  system  that  will  duplicate  the  current  system’s 
functionality  with  minimal  rewriting  of  our  software.  Our  interest  in  developing  this  alternative 
display  workstation  is  both  to  provide  a  backup  for  our  current  system  and  to  make  a  stereo 
display  system  available  to  the  mammographers  at  UMMC  where  the  stereo  images  are  being 
acquired. 

5.  Identification  and  Development  of  Stereo-based  Visual  Features  (Tasks  5,6,7) 

Our  plan  for  meeting  this  objective  called  for  the  completion  of  three  Tasks  (5,  6  and  7). 
Task  5  required  conducting  individual  interviews  with  our  three  expert  mammographers  as  they 
closely  studied  a  representative  set  of  stereo-imaged  cases  to  evaluate  them  in  general  and  to 
identify  all  promising  new  features  of  stereo-based  diagnostic  information.  Task  6  required 
conducting  a  consensus  meeting  of  the  three  experts  to  develop  an  explicit  description  and  final 
listing  of  all  identified  features.  Task  7  required  developing  scales  for  those  features  and 
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assembly  of  a  new  feature  checklist/questionnaire  to  be  employed  in  the  Final  Reading  Study. 

The  plan,  revised  in  Year  2  to  reflect  delays  in  case  accrual,  called  for  all  three  of  the  these  tasks 
to  be  completed  by  the  end  of  Year  3. 

We  accomplished  a  first  round  of  interviews  with  the  three  experts  earlier  in  Year  3. 

Then,  based  on  the  experts’  general  evaluations  of  the  stereo  display,  particularly  their 
uncertainty  about  how  well  calcification  clusters  could  be  rendered  in  the  digital  soft-copy 
display,  we  decided  to  interrupt  Task  5  to  conduct  a  Preliminary  Reading  Study.  The  point  of  the 
preliminary  study  was  to  give  us  a  better  sense  of  the  stereo  display’s  adjunctive  effectiveness  in 
diagnosing  each  of  the  three  main  types  of  lesion,  as  well  as  to  give  the  experts  some  intensive 
experience  at  reading  with  the  stereo  displays  before  proceeding  further  with  Task  5.  The  plan 
now  is  to  complete  Task  5,  and  to  go  on,  in  short  order,  to  completion  of  Tasks  6  and  7  early  in 
Year  4. 

5.1  Interviews  and  studies  with  mammography  experts  (Task  5) 

Each  expert  was  interviewed  individually.  The  general  plan  was  to  conduct  the 
interviews  in  two  stages:  (1)  an  open-ended  interview  that  encouraged  the  experts  to  express  their 
thoughts  in  whatever  terms,  and  with  whatever  emphasis,  seemed  fitting  to  them  individually, 
and  (2)  a  guided  interview  that  allowed  us  to  bring  to  the  discussion  the  ideas  and  terminology  of 
the  other  two  experts.  The  original  plan  was  to  conduct  the  second  stage  of  interviewing  as  soon 
as  we  had  completed  the  interviews  of  all  three  experts  in  the  first  stage.  We  then  decided  to 
insert  the  Preliminary  Reading  Study  in  between  the  two  stages. 

Open-ended  interviews.  The  general  approach  was  to  have  the  experts  read  the 
conventional  films  and  stereo  images  of  each  of  a  representative  set  of  cases  of  each  lesion  type. 
They  were  asked  to  think  aloud  as  they  read  each  case,  to  identify  and  describe  all  features  of 
diagnostic  interest,  particularly  those  newly  available  in  the  stereo  display,  or  where  the  stereo 
display  made  a  significant  adjunctive  clarification  of  features  perhaps  only  weakly  visible  in  the 
films.  They  were  also  asked  to  evaluate  the  stereo  display  in  more  general  terms,  to  express  any 
and  all  thoughts  on  its  strengths  and  weaknesses  for  mammography. 

To  prepare  the  experts  for  these  interviews,  we  first  reviewed  with  them  the  technology  of 
the  stereo  imaging,  how  the  images  are  obtained  and  displayed.  We  also  explained  the  nature  of 
the  display  interface  and  had  each  expert  operate  the  system  and  practice  with  the  various  options 
for  manipulating  the  stereo  images. 

We  then  systematically  reviewed  the  nature  of  the  volumetric  information  potentially 
retrievable  from  these  displays — ^with  respect  to  a  lesion’s  locus  and  orientation  in  depth  in  the 
breast  parenchyma,  with  respect  to  its  volumetric  shape,  with  respect  to  the  layout  and 
arrangement  in  depth  of  its  constituent  parts,  and  with  respect  to  size,  shape  and  orientation  in 
depth  of  the  individual  parts — ^potentially  an  enormous  amount  of  new  diagnostic  data.  Some  of 
this  information,  of  course,  like  the  volumetric  shape  of  a  mass,  would  not  be  entirely  new,  in  the 
sense  that  it  is  retrievable,  at  least  in  a  crude  fashion,  by  merging  information  retrieved  separately 
from  the  two  films.  But  even  so,  improvement  in  comprehension  and  precision  of  that 
information  in  the  stereo  display  might  well  support  new  nuances  of  discrimination  and 
interpretation  not  feasible  with  the  films.  Our  aim  in  this  review  was  to  encourage  the  experts  to 
consider  all  of  these  new  possibilities.  Some  of  the  information  provided  by  the  stereo  display 
would,  of  course,  potentially  open  entirely  new  doors  of  analysis  and  interpretation,  and  we 
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encouraged  the  experts  particularly  to  consider  those  possibilities.  To  illustrate  this  point,  we 
brought  specifically  to  their  attention  new  options  that  the  stereo  display  provides  for  analyzing 
calcification  clusters.  In  the  films  there  is  no  way  for  the  mammographer  to  discern  the  actual 
arrangement  in  depth  of  the  individual  elements.  This  is  because  in  the  films  there  is  no  way  to 
reliably  pair  an  element  in  one  view  with  itself  in  the  other.  The  stereo  enables  pairing  every 
element  with  itself  and  thereby  reveals  details  of  structure  not  retrievable  from  the  films.  We 
pointed  out  similar  new  possibilities  with  respect  to  intricacies  of  the  arrangement  of  linear 
elements  comprising  architectural  distortion,  where,  again,  there  is  no  way  to  pair  up  the 
elements  seen  separately  in  the  films. 

Then,  working  with  a  representative  set  of  six  benign  and  six  malignant  cases  of  each 
lesion  type  (mass,  calcification  cluster  and  architectural  distortion),  we  had  each  expert  conduct  a 
careful  examination  of  the  stereo  images.  Each  case  was  complete  with  the  full  set  of  stereo 
images,  diagnostic  films  and  pathology  report.  Taking  into  consideration  the  pathology  truth  data 
and  the  appearance  of  the  lesion  on  the  films  on  which  the  lesion  was  diagnosed,  the  expert’s 
task  was  to  consider  every  aspect  of  appearance  of  the  lesion  in  the  stereo  display  that  helped  to 
advance  the  accuracy  of  diagnosis  beyond  what  would  have  been  achieved  by  conventional  film 
mammography  alone,  either  by  strengthening  what  might  have  been  a  true  diagnosis  or 
weakening  what  might  have  been  a  false  one.  The  experts  were  asked  to  think  aloud  as  they  read 
the  stereo  images,  made  back  and  forth  comparisons  to  the  conventional  films  and  considered  the 
pathology  truth  data.  We  recorded  their  comments,  interacted  with  them  to  clarify  particular 
points,  and  developed  an  overview  of  their  thoughts  by  lesion  type. 

For  masses,  their  comments  centered  mainly  on  how  the  stereo  display  helped  to  separate 
the  mass  from  extraneous  dense  tissue  lined  up  with  it  in  the  volume,  but  located  elsewhere  in 
depth.  They  commented  on  how  that  separation  helped,  in  turn,  in  determining  the  shape  of  the 
mass  and  also  in  differentiating  mere  obscuration  of  the  boundary  by  those  unrelated  but 
overlapping  densities,  as  opposed  to  truly  worrisome  indistinctness  of  the  boundary.  They  also 
commented  on  how  the  stereo  display  helped  them  to  discern  spiculations  emanating  from  the 
mass  and  to  detect  protrusions  that  were  not  evident  from  analysis  of  the  shapes  of  the  two  film 
views. 


For  calcification  clusters,  the  comments  centered  on  two  main  contributions  of  the 
stereo  display.  First,  were  comments,  akin  to  those  with  respect  to  masses,  on  how  the  stereo 
display  helped  in  separating  the  cluster  of  interest  from  elements  of  calcification  that  were  lined 
up  with  it  in  the  volume,  but  located  elsewhere  in  depth.  Second,  were  comments  on  how  the 
stereo  display  provides  a  compelling  sense  of  the  volumetric  arrangement  of  the  elements 
comprising  the  cluster  and  how  this  helps  in  a  variety  ways  to  increase  the  accuracy  of  diagnosis. 
In  one  case,  a  cluster  seen  on  the  conventional  films  as  large  and  worrisome  could  be  seen  in  the 
stereo  images  as  two  separate  and  benign  appearing  clusters.  In  another  case,  elements  that 
appeared  in  the  conventional  films  to  be  distributed  randomly  in  the  cluster  could  be  seen  in  the 
stereo  display  as  in  a  benign,  dome-like  arrangement.  The  experts  also  reported  repeatedly  on 
how  the  stereo  display  was  helpful  in  revealing  linearity  and  branching  not  evident,  or  not  as 
clearly  evident,  in  the  conventional  films.  The  experts  also  commented  on  how  the  stereo  display 
helped  to  see  large-scale  arrangements  of  calcifications,  to  detect  worrisome  segmental  patterns, 
and  to  integrate  more  confidently  the  extended  collections  of  benign  vascular  calcifications. 

For  architectural  distortion,  the  experts’  comments  also  centered  on  the  stereo  display’s 
help  in  separating  components  of  the  lesion  from  extraneous  material  lined  up  with  it  in  the 
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volume  but  located  elsewhere  in  depth,  and  to  detect  the  presence  of  a  subtle  central  focus  that 
could  be  missed  on  film.  Here  the  components  are  the  strands  of  dense  tissue  that  appear  to 
radiate  from  a  central  point.  The  comments  also  suggest  that  the  stereo  display  helps  to  determine 
whether  the  strands  that  are  seen  to  be  at  the  same  level  in  depth  also  originate  at  the  same  locus 
in  depth. 

While  these  specific  reactions  to  the  stereo  display  were  all  quite  positive,  the  experts  did 
convey  two  general  concerns  about  inherent  limitations  of  the  stereo  display.  Their  primary 
concern  was  with  respect  to  lower  spatial  resolution  of  the  stereo  display  compared  to  the 
conventional  films,  a  matter  most  apparent  to  them  in  often  not  being  able  to  see  all  of  the 
calcifications  visible  on  the  conventional  films.  This  lack  of  resolution  limited  their  ability  to  see 
the  full  arrangement  of  the  elements.  It  also  degraded  their  ability  to  see  clearly  the  orientations 
and  shapes  of  those  individual  elements  that  were  visible.  Their  second  concern,  and  one,  again, 
that  seemed  to  them  most  significant  with  respect  to  calcifications,  was  the  aspect-ratio  distortion 
inherent  in  the  current  stereo  display  configuration.  The  vertical  stretching  of  the  stereo  display 
had  the  effect  of  distorting  the  shapes  of  the  masses,  the  distribution  of  the  calcification  clusters 
and  the  shapes  of  the  individual  elements  of  calcification.  While  to  some  extent  the  experts 
could  be  expected  take  those  known  distortions  into  account,  it  is  clearly  not  something  that  is 
fully  amenable  to  mental  factoring,  and  the  experts  found  it  particularly  deleterious  with  respect 
to  assessing  the  shapes  of  the  elements  of  calcification.  These  concerns  led  them  and  us  to  some 
significant  uncertainty  about  just  how  effective  the  stereo  display  would  be,  despite  the  many 
promising  ways  in  which  it  could  potentially  help,  and  particularly  in  regard  to  its  effectiveness 
in  diagnosing  calcification  clusters.  As  a  consequence,  we  decided  it  would  be  wise  to  do  a 
Preliminary  Reading  Study  just  to  see  where  we  stood.  We  also  sought  the  benefit,  before  going 
further  with  the  feature  explorations  of  Task  5,  of  having  the  experts  do  some  intensive  reading 
with  the  stereo  display — ^intensive  in  the  sense  of  their  being  blinded  to  the  truth  and  scored  for 
accuracy. 

Debriefings  of  the  experts  and  analyses  of  the  logs  of  their  case-by-case  comments 
following  the  Preliminary  Reading  Study.  The  Preliminary  Reading  Study  had  two  main 
purposes.  One  was  to  obtain  a  rough  indication  of  the  stereo  display’s  adjunctive  contribution  to 
diagnostic  accuracy.  This  was  to  provide  guidance,  in  light  of  the  suggested  weaknesses  of  the 
stereo  display,  particularly  for  conveying  information  on  calcifications,  on  the  possible  need  to 
adjust  our  approach  to  the  Final  Reading  Study.  The  other  aim  was  to  expand  our  effort  in  Task 
5 — to  explore  further  for  features  newly  available  or  significantly  clarified  with  the  stereo 
display,  and  to  assess  further  the  general  effectiveness  of  stereo  as  an  adjunctive  contributor  to 
mammography.  The  design  of  the  Preliminary  Reading  Study,  the  overall  results  and  their 
implications  for  our  approach  to  the  Final  Reading  Study  are  given  below  in  Section  6.  Here  we 
describe  contributions  of  the  Preliminary  Reading  Study  to  the  aims  of  Task  5. 

We  interviewed  each  of  the  four  readers  at  points  during  the  scheduled  readings  and  after 
they  had  completed  the  study.  The  debriefings  were  generally  consistent  with  the  findings  of  the 
open-ended  interviews — recognition  of  stereo’s  several  contributions  and  a  voicing  of  some 
fundamental  concerns.  One  concern  brought  up  anew  in  these  interviews  was  in  regard  to  the 
limited  brightness  and  contrast  of  the  display  monitor.  The  concern  was  that  it  was  much  lower 
than  what  can  be  achieved  with  the  films,  limiting  the  gray  scale  distinctions  that  can  potentially 
be  made  with  the  stereo  and  also  probably  a  big  factor  in  the  loss  of  visibility  of  small 
calcifications. 
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We  also  had  the  readers  keep  a  log  of  comments  on  a  case-by-case  basis.  The  case-by¬ 
case  comments  were  also  generally  consistent  with  what  we  learned  in  the  open-  ended 
interviews.  The  specific  comment  on  many  of  the  cases  was  that  the  stereo  display  did  not  show 
calcifications  as  well  as  the  films  did,  though  on  occasion,  the  stereo  showed  calcifications  not 
visible  on  the  films.  The  stereo  display  was  repeatedly  found  to  help  in  interpreting  masses  - 
improving  the  visibility  of  spiculations  and  interior  fat,  clarifying  the  nature  of  the  boundary, 
helping  to  localize  and  associate  calcifications  with  the  mass,  and  clarifying  associated 
architectural  distortion.  Despite  the  shortcoming  of  not  showing  all  of  the  calcifications  visible 
on  the  films,  the  stereo  display  was  repeatedly  reported  to  clarify  how  the  elements  of  a  cluster 
were  arranged.  While  in  a  few  cases,  architectural  distortion  evident  in  the  films  was  not  visible 
in  the  stereo,  generally  stereo  was  found  to  bring  out  architectural  distortion. 

The  improved  understanding  of  the  strengths  and  weaknesses  of  the  stereo  display  that 
the  Preliminary  Reading  Study  has  provided  in  these  various  comments  puts  us  in  a  much  better 
position  to  conduct  the  guided  interviews.  Also,  of  our  three  experts,  only  Dr.  D’Orsi  had  any 
prior  experience  reading  the  stereo  displays.  So,  Drs.  Meyer  and  Sadowsky,  having  participated 
in  the  intensive  reading  experience  and  critique  of  the  Preliminary  Reading  Study,  are  now  much 
better  equipped  to  proceed  to  the  guided  interviews  and  into  the  consensus  meeting. 

Guided  Interviews.  To  complete  Task  5,  we  will  conduct  a  guided  interview  of  each  our 
three  experts  early  in  Year  4.  We  will  use  as  the  primary  guide  the  old  version  of  our 
mammography  checklist/questionnaire  developed  for  use  in  diagnosing  lesions  seen  on  films 
alone.  The  general  approach  will  be  to  walk  through  the  old  questionnaire,  considering  with  the 
expert  how  each  of  the  features  might  appropriately  be  modified  to  reflect  the  expanded 
perception  and  knowledge  of  the  features  afforded  by  addition  of  the  stereo  information.  For  each 
of  the  old  features,  we  will  look  for  descriptions  and  terminology  that  are  sensitive  to  the  newly 
accessible  volumetric  information.  We  will  also  probe  carefully  for  the  possibility  of  entirely 
new  features,  emergent  properties  that  are  perhaps  only  visible  in  stereo.  After  completing  these 
guided  interviews,  we  will  circulate  the  questionnaires,  with  the  suggested  changes  highlighted, 
to  the  other  experts  for  their  review  prior  to  the  consensus  meeting. 

5.2  Consensus  meeting  (Task  6) 

At  the  consensus  meeting,  we  will  systematically  review  the  various  changes  and 
additions  to  the  old  questionnaire  that  were  suggested  in  the  guided  interviews  of  Task  5  and 
work  to  achieve  consensus  on  those  that  will  be  incorporated  in  the  new  version. 

5.3  Construct  new  Checklist/questionnaire  (Task  7) 

In  constructing  the  new  checklist/questionnaire,  we  will  draw  heavily  on  the  design  of  the 
old  one.  Those  features  that  remain  unchanged  will  be  incorporated  directly  into  the  new 
questionnaire.  Where  the  wording  of  a  question  has  been  changed,  we  will  review  carefully 
whether  the  changed  wording  requires  changes  in  the  design  of  the  reference  rating  scale  and 
make  changes  accordingly,  e.g.  change  the  wording  or  graphics  of  the  anchor  points  on  the  scale. 
We  will  also  utilize  the  same  overall  arrangement  and  formatting  as  on  the  old  questionnaire, 
unless  any  of  the  new  questions,  or  any  of  the  changes  to  old  questions,  require  us  to  depart  from 
that. 

6.  Design,  Conduction  and  Analysis  of  a  Preliminary  Reading  Study  (Tasks  10, 1 1, 12) 
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6.1  Design  of  the  Study 


Cases.  The  case  set  for  the  preliminary  study  consisted  of  76  cases  with  pathology- 
proven  lesions.  For  each  of  these  cases,  we  had  acquired  the  set  of  diagnostic  films  that  had  led 
to  the  biopsy  of  the  lesion.  The  distribution  of  lesions  by  type  and  by  pathology  truth  is  shown 
below  in  Table  2. 


Table  2.  Distribution  of  lesions  by  type  and  pathol 

‘ogy- 

Lesion  Type 

Benign 

Truth 

Malignant 

Total 

Mass 

22 

12 

34 

Calcifications 

29 

4 

33 

Architectural 

Distortion 

2 

7 

9 

Total 

53 

23 

76 

Readers.  Four  highly  experienced  mammographers  participated  in  this  study.  Two  of 
these  mammographers  were  two  of  the  three  expert  consultants  to  the  project,  namely.  Dr.  Jack 
Meyer  and  Dr.  Norman  Sadowsky.  The  other  two  mammographers  were  Dr.  Larry  Moss  from 
the  University  of  Massachusetts  Medical  Center  and  Dr.  Elsie  Levin  from  Faulkner  Hospital. 

Study  design.  Each  reader  participated  in  two  reading  sessions  (each  approximately  a 
half  day  in  length)  reading  half  of  the  cases  in  the  first  session  and  the  other  half  of  the  cases  in 
the  second.  The  cases  were  read  without  knowledge  of  pathology  truth,  and  none  were  ones  that 
the  two  experts  had  previously  seen,  with  pathology  truth,  in  the  guided  interview.  Each  set  was 
arranged  in  a  single  random  order.  At  the  start  of  the  first  reading  session,  we  provided  a  brief 
training  session  in  which  the  reader  was  familiarized  with  the  operation  of  the  stereo  display 
workstation  and  given  an  opportunity  to  practice  with  its  keyboard  controls. 

Each  case  was  read  in  three  phases,  in  immediate  succession.  In  the  first  phase,  the 
reader  was  shown  the  set  of  available  film  images  from  the  diagnostic  study  (mounted  on  an 
automated  film  viewer)  and  told  the  patient’s  age.  The  reader  was  then  required  to  estimate  the 
probability,  on  a  100-point  scale,  that  the  lesion  was  malignant  and  to  enter  that  value  via  the 
keyboard  for  recording  by  the  system. 

In  the  second  phase  of  reading  a  case,  the  reader  could  view  one  or  the  other  of  the  two 
available  digital  images  for  the  case  in  non-stereo  mode  on  the  display  monitor.  In  this  phase, 
only  non-stereo  viewing  of  the  images  was  allowed;  both  eyes  always  saw  the  same  image.  The 
reader  was  allowed  to  switch  between  the  two  images,  but  was  only  able  to  see  one  at  a  time.  The 
films  for  the  case  were  still  available  to  the  reader  for  reference.  The  reader  was  also  able  to 
control  aspects  of  the  display,  including  brightness,  contrast,  and  grayscale  inversion.  After 
viewing  the  non-stereo  presentations  of  the  digital  images,  the  reader  was  required  to  make  a  new 
assessment  of  the  probability  that  the  lesion  was  malignant,  taking  into  account  the  information 
available  both  on  the  films  and  on  the  non-stereo  display,  and  to  key  that  assessment  into  the 
system. 
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In  the  third  phase,  the  display  was  enabled  to  show  the  reader  the  two  digital  images  as  a 
single  stereoscopic  pair.  The  reader  put  on  the  stereo-viewing  eyeglasses  (StereoGraphics 
CrystalEyes  LCD  glasses)  to  view  the  images  in  this  stereoscopic  mode.  The  case’s  films 
remained  available  to  the  reader  for  reference.  The  reader  was  able  to  control  aspects  of  the 
stereo  display,  including  brightness,  contrast,  grayscale  inversion,  plus,  in  this  condition,  depth 
inversion.  After  viewing  the  stereo  image,  the  reader  again  was  required  to  assess  the  probability 
that  the  lesion  was  malignant,  this  time  taking  into  account  the  additional  information  provided 
by  seeing  the  lesion  in  stereo. 

6.2  Results  of  the  Study 

Accuracy  analyses.  We  conducted  ROC  analyses  on  the  pooled  data  of  our  4  readers. 

We  used  as  our  measure  of  diagnostic  accuracy,  Az,  the  area  under  the  fitted  ROC  curve.  The 
estimated  values  of  Az  for  film  alone,  film  +  non-stereo  digital,  and  film  +  stereo  digital  were 
0.73, 0.72  and  0.76.  The  film-alone  accuracy  of  0.73  is  relatively  low  for  standard 
mammography,  indicating  that  this  case  set  is  relatively  difficult  to  diagnose.  Statistical  analysis 
of  the  results  showed  that  film  and  film  +  non-stereo  digital  were  not  significantly  more  accurate 
than  film  alone  (p>.83).  On  the  other  hand,  film  +  stereo  digital  was  significantly  more  accurate 
than  film  alone  (p<.04).  Thus,  the  stereo  mammogram  added  a  significant  amount  of  adjunctive 
diagnostic  information  to  that  obtained  from  the  films  alone.  Furthermore,  we  believe  that 
several  limitations  of  the  digital  images  and  of  the  digital  display,  discussed  below  in  our  analysis 
of  the  reader  logs,  have  resulted  in  an  underestimation  of  the  potential  adjunctive  accuracy 
afforded  by  the  stereo  display. 

We  would  have  liked  to  examine  the  accuracy  results  broken  down  by  type  of  lesion. 
However,  we  were  unable  to  carry  out  this  analysis  because  the  number  of  cases  in  each  of  the 
resulting  subsets  is  too  small  to  obtain  reliable  results.  We  note  that  we  will  be  able  to  conduct 
this  type  of  analysis  on  the  results  of  the  final  reading  study,  where  we  will  have  available  a 
much  larger  case  set. 

Analysis  of  reader  logs.  We  have  read  carefully  through  the  log  of  comments  written  out 
by  each  reader  as  he/she  read  cases.  From  their  comments,  we  can  draw  several  general 
conclusions,  as  well  as  observations  about  particular  cases.  The  most  important  conclusion, 
commented  on  by  all  four  readers,  is  that  the  spatial  resolution  of  the  digital  images  (whether 
stereo  or  non-stereo)  is  poorer  than  that  of  the  films.  This  is  manifest  largely  in  cases  presenting 
with  small,  fine  micro-calcifications.  Our  readers  frequently  conunented  that  some  of  the 
calcifications  visible  in  the  films  were  simply  not  visible  in  the  digital  images.  (However,  we 
should  note  also  that  occasionally  readers  remarked  that  calcifications  were  more  clear  or  more 
visible  in  the  digital  images  than  on  the  film).  We  believe  that  there  may  be  two  reasons  for  the 
apparently  poorer  spatial  resolution  in  the  digital  images.  The  first  is  a  possible  limitation  of  the 
100-micron  spot  size  of  the  GE  digital  mammography  unit.  There  is  some  reason  to  think  that 
100  microns  is  too  large  a  spot  size  to  adequately  capture  very  small  calcification  elements.  A 
second  reason — ^perhaps  the  more  important  one  here — is  a  degradation  of  available  spatial 
resolution  by  the  shadow  mask  in  the  color  display  monitor  used  in  the  stereo  display 
workstation.  The  fact  that  each  grayscale  pixel  on  our  color  monitor  necessarily  is  composed  of 
triads  of  equally  illuminated  color  dots,  guarantees  some  loss  of  spatial  resolution  available  in  the 
image.  We  think  that  this  problem  is  sufficiently  serious  that,  before  we  conduct  the  final 
reading  study,  we  hope  to  replace  the  color  monitor  with  a  high-resolution  grayscale 
(monochrome)  monitor. 
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A  second  general  observation  was  that  vertical  elongation  present  in  the  display — an 
uncorrectable  distortion  characteristic  of  this  monitor — sometimes  created  a  problem  in 
interpreting  a  displayed  lesion.  The  distortion  arises  because  in  order  to  present  a  correct  aspect 
ratio,  the  vertical  size  of  the  displayed  image  needs  to  be  half  of  the  screen  height  rather  than  full 
height.  The  autosync  circuits  in  this  (and,  in  fact,  in  most)  monitors  attempt  to  automatically 
cause  the  image  to  fill  the  screen  vertically.  Unfortunately,  the  manual  vertical  size  control  on 
our  Hitachi  monitor  will  only  permit  reduction  of  the  vertical  image  size  to  about  two-thirds  of 
the  full  screen  height,  causing  a  displayed  circular  area  to  appear  as  an  ellipse,  elongated 
vertically.  While  our  readers  could  take  this  known  distortion  into  account  fairly  well,  it  did 
make  it  difficult  to  decide,  sometimes,  whether  an  apparently  linear  calcification  element  was 
truly  linear  or  only  optically  stretched.  Similarly,  a  mass  that  was  truly  spherical  would  take  on 
the  appearance  of  an  ovoid.  We  regard  this  as  a  serious  problem,  too,  and  will  hope  to  solve  it 
when  we  find  an  appropriate  grayscale  monitor. 

Perhaps  the  most  impressive  finding  from  our  review  of  the  case-by-case  comments  was 
that  in  10  of  the  76  cases  read,  an  additional  lesion  was  detected  in  the  stereo  display  by  one  or 
more  readers  that  could  not  be  seen  in  the  films.  In  3  of  the  cases,  a  mass  was  visualized  that 
was  not  evident  on  the  film.  In  another  3  cases,  architectural  distortion  was  seen  on  the  stereo 
image  but  not  on  the  film.  And  in  the  other  4  cases,  (additional)  calcifications  were  detected  that 
were  not  visible  on  the  film.  This  finding  suggests  that  a  significant  role  for  stereo 
mammography  may  lie  in  increasing  the  detectability  of  early,  subtle  lesions  that  are  not  visible 
in  standard  two-dimensional  films.  While  the  increased  usage  of  needle  and  large-core  biopsies  is 
currently  reducing  the  need  for  improved  mammographic  characterization  of  lesions,  that 
advantage  holds  only  if  the  lesion  is  detected  and  localized.  Stereo  mammography  may  offer 
substantial  aid  towards  improved  detection. 
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7.  Level  of  Effort  and  Schedule 


The  project  ran  at  a  level  of  effort  and  expenditure  through  Year  3  of  close  to  100%  of 
budget.  All  staff  continue  with  the  project  as  planned.  One  of  our  three  expert  mammography 
consultants.  Dr.  Thomas  Frenna,  left  his  position  on  the  mammography  staff  of  Brigham  and 
Women’s  Hospital  (BWH)  in  Boston  last  year  and  moved  to  a  practice  in  New  York.  We  are 
pleased  that  Dr.  Norman  Sadowsky,  Director  of  the  Faulkner-Sagoff  Centre  at  Faulkner  Hospital, 
agreed  to  replace  Dr.  Frenna  on  the  project  as  an  expert  consultant  on  mammography. 

Because  our  expert  mammographers  had  no  prior  experience  with  viewing  and 
interpreting  stereo  mammograms,  our  progress  in  developing  a  checklist  of  visual  features, 
perceptible  in  stereo,  has  been  slower  than  we  had  hoped.  In  order  to  give  them  some  intensive 
experience  looking  at  stereo  mammograms,  we  decided  to  run  the  Preliminary  Reading  Study 
described  above.  We  will  now  complete  the  feature  and  checklist-development  work  early  in 
Year  4  and  move  on  to  the  Final  Reading  Study.  Consequently,  we  are  proposing  to  modify  the 
timeline  as  shown  below  in  Table  3.  The  work  to  be  completed  within  the  set  of  project  tasks 
remains  as  originally  proposed;  only  the  timeline  for  when  several  of  the  tasks  begin  and  end  has 
been  modified. 


Table  3.  Revised  Timeline  and  Status  of  Project  Tasks 


Task# 

Original 

Period 

New  Revised 
Period 

Status 

Task  Description 

Taskl 

Months  1-8 

Months  1-12 

Completed  * 

ile6f|amze  user  interface  ; 

Task  2 

Months  9-18 

Months  1-18 

Completed  * 

Add  new  functionality  to  cotttipl  ' 

Task  3 

Months  1-18 

Months  1-24 

Completed  * 

Explore  joystick  and  speech  control 

Task  4 

Months  1-36 

Months  11-48 

Ongoing 

Acquire  stereo-imaged  cases 

Tasks 

Months  6-11 

Months  18-41 

Mostly 

completed 

Interviews  and  studies  witl^.maramo.  experts 

Task  6 

Month  12 

Month  42 

To  be  done 

"SdnsWainieetihg  vffij  mahottho.  v. : 

Task? 

Months  13-18 

Months  30-42 

To  be  done 

Develop  checklist  of  s®eo  featihes 

Tasks 

Months  19-24 

Montiis  19-32 

Completed* 

Explore  Stereo  displayjmddes  witfi  exi^rts 

Task  9 

Months  25-30 

Months  25-36 

Completed* 

Modify  MVplay  software^or  new  modes 

Task  10 

Months  25-30 

^’|yl0htl£:36-43  -  I 

Prelini  study 
completed 

forfinal  study  '  > 

;^Task:||;.,^ 

i 

Months  31-42 

llllllto  . 

Prelim  study 
completed 

Conduct  jSnal  reading  study . 

-  '-4  <  •  "■ 

Task  12 

Months  43-48 

Months  46-48 

Prelim  study 
completed 

Analyze  final  reading  study  results 

Task  13 

Months  43-48 

Months  46-48 

To  be  done 

Train  statistical  predicMon  rules  on  feature 
data  and  measure  accuracies 

*  Work  originally  proposed  within  these  tasks  has  been  completed;  however,  we  will  continue  to 
work  on  them  as  needs  arise  within  the  project. 


8.  Plans  for  Year  4 

As  indicated  by  the  proposed  timeline  shown  above,  we  will  continue  accruing  cases 
(Task  4)  throughout  Year  4. 
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Over  the  early  months  of  Year  4,  we  will  conduct  guided  interviews  with  our  three 
manunography  experts,  to  identify  visual  features  perceptible  in  the  stereo  images  (Task  5).  This 
activity  will  culminate  in  a  consensus  meeting  with  the  three  experts  in  Month  42  to  arrive  at  a 
master  feature  list  (Task  6).  As  we  identify  these  features,  we  will  construct  the 
checklist/questionnaire  for  use  in  the  Final  Reading  Study  (Task?). 

Finally,  in  the  third  quarter  of  Year  4,  we  will  conduct  the  Final  Reading  Study  using  all 
cases  acquired  at  that  time  (Tasks  10, 11).  In  the  final  quarter,  we  will  analyze  the  results  of  the 
study  and  train  statistical  prediction  rules  using  the  feature  data  provided  by  the  readers  (Tasks 
12, 13). 
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KEY  RESEARCH  ACCOMPLISHMENTS 


•  Acquired  166  stereo-imaged  lesions  at  end  of  Year  3. 

•  Developed  semi-automated  software  to  aid  in  processing  of  raw  digital  stereo  images  for 
viewing. 

•  Completed  development  of  new  application  software  that  improves  the  user  interface  of  the 
stereo  display  workstation. 

•  Conducted  interviews  with  three  expert  mammographers  to  explore  and  understand  new 
information  visible  in  stereo  mammograms. 

•  Designed,  conducted  and  analyzed  a  Preliminary  Reading  Study  (4  mammographers,  76 
cases)  comparing  film  alone,  film  plus  non-stereo  digital  images,  and  film  plus  stereo  digital 
images. 

•  In  the  Preliminary  Reading  Study,  we  obtained  a  statistically  significant  gain  in  diagnostic 
accuracy  from  stereo  versus  film  alone. 

•  In  the  Preliminary  Reading  Study,  additional  lesions — ^not  visible  on  the  films — ^were 
detected  on  the  stereo  mammogram,  in  10  of  the  76  cases. 

REPORTABLE  OUTCOMES 

Dr.  Getty  presented  the  research  efforts  of  this  project  at  the  Eighth  Far  West  Image 
Perception  Conference  held  at  the  Nakoda  Lodge  in  Alberta,  Canada  on  May  28-30, 1999.  In 
particular,  he  described  the  results  of  the  preliminary  reading  study  to  the  meeting,  in  a  talk 
entitled  “Stereoscopic  Digital  Mammography:  What  Can  You  See  in  a  Stereo  View  of  the  Breast 
That  You  Can’t  See  in  Two  Standard  Orthogonal  2D  films?”  Textual  slides  from  this  talk  are 
included  as  Appendix  B  of  this  report.  Dr.  Getty  also  set  out  an  exhibit  at  the  meeting  consisting 
of  six  sample  stereo  mammograms  captured  on  35mm  slides  and  presented  in  hand-held 
stereoscopic  viewers.  Meeting  participants  were  invited  to  come  view  the  stereo  mammograms 
at  their  leisure  over  the  next  several  days. 

CONCLUSIONS 

We  have  four  main  conclusions  to  draw  from  the  work  accomplished  to  date. 

First,  we  have  initial  evidence  that  stereo  can  improve  accuracy  in  diagnosing  breast 
cancer.  Even  with  a  relatively  small  number  of  test  cases,  our  carefully  controlled  Preliminary 
Reading  Study  found  a  statistically  significant  adjunctive  improvement  in  diagnostic  accuracy 
using  the  stereo  display.  That  gain  is  relatively  small,  but  still  very  encouraging  considering  that 
the  stereo  display,  as  implemented  in  this  project,  has  the  two  significant  but  correctable 
limitations  described  below.  A  much  larger  gain  can  be  expected  when  those  limitations  are 
corrected. 

Second,  stereo  has  a  strong  potential  for  improving  cancer  detection.  Very  important,  in 
our  view,  was  the  finding  in  our  case-by-case  analyses  that  for  ten  different  cases  the 
mammographers  detected  additional  lesions  in  the  stereo  display  that  were  not  visible  at  all  on 
the  films.  This  suggests  that  stereo  has  a  very  important  potential  for  cancer  detection.  The 
current  strategy  of  increasing  the  accuracy  of  detection  by  relying  on  the  biopsying  of  more  and 
more  marginal  radiographic  evidence  can  not  protect  the  patient  in  situations  where  even 
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marginal  evidence  of  a  tumor  is  absent  the  films.  Stereo  apparently  could  have  a  very  important 
role  to  play  in  catching  many  of  those  cases. 

Third,  stereo  is  likely  to  be  easily  and  enthusiastically  accepted  by  mammographers  in 
the  field.  All  of  our  readers  were  very  intrigued  by  the  stereo  display  and  quickly  found  it  easy  to 
control  the  system  and  interpret  the  images.  The  stereo  display  provides  a  way  for  them  to  derive 
volumetric  information  clearly  and  validly  via  an  effortless  perceptual  process,  something  they 
can  only  struggle  to  do  crudely  by  looking  back  and  forth  between  the  two  film  views.  Yet,  the 
stereo  display  has  all  same  familiar  feel  as  the  films  and  so  brings  no  additional  burden  to  the 
interpretive  process. 

Fourth,  we  have  found  that  the  version  of  stereo  employed  in  this  study  suffers  two  main, 
but  correctable  limitations.  The  first  main  limitation  is  with  respect  to  the  resolution  and 
brightness  contrast  achievable  with  the  present  display.  It  shows  much  less  of  the  fine  detail 
visible  on  the  films  than  we  had  expected  it  would.  We  are  confident  though  that  this  problem 
can  be  corrected  with  a  better,  grayscale-only  monitor.  The  second  main  problem  is  that  because 
of  a  limitation  in  adjustability  of  the  present  monitor’s  vertical  image  height,  the  display  distorts 
the  shapes  of  lesions  and  lesion  components.  The  readers  have  found  that  it  is  difficult  to  factor 
that  distortion  into  their  thinking,  and  that  it  hinders  their  interpretations.  We  are  looking  into 
obtaining  a  new  monitor  that  can  correct  both  of  these  limitations  before  proceeding  to  the  Final 
Reading  Study. 
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Appendix  A:  User's  Manual  -  Stereoscopic  Mammography  Workstation 


User’s  Manual 

Stereoscopic  Mammography  Workstation 


1.0  Introduction 

The  Stereoscopic  Mammography  Workstation  is  a  Visual  Basic  application  for  displaying  a 
pair  of  digitized  x-ray  images  stereoscopically.  It  gives  the  user  control  over  the  brightness 
and  contrast  of  the  stereo  image  as  well  as  other  aspects  of  the  stereo  display  through  two 
types  of  interfaces: 

•  A  Graphical  User  Interface  (GUI  Mode) 

•  A  mouse  interface  (Mouse  Mode). 

The  GUI  mode  provides  the  user  with  standard  graphical  controls  such  as  menus  and 
buttons  to  execute  commands.  However,  the  GUI  is  not  convenient  for  executing 
commands  that  control  the  stereo  display  for  two  reasons: 

•  The  special  eyeglasses  required  for  viewing  the  stereo  display  are  problematic  when 
viewing  the  GUI  screen  because  of  flicker. 

•  The  user  has  to  look  away  from  the  stereo  display  while  making  changes  to  the  stereo 
image  from  the  GUI. 

Consequently,  a  second  mode  of  interaction.  Mouse  Mode,  is  provided  for  making  changes 
to  the  stereo  (tisplay.  In  this  mode,  the  user  can  focus  solely  on  the  stereo  image  while 
moving  the  mouse  or  clicking  mouse  buttons  to  enhance  it. 

1.1  GUI  Mode 

The  application  provides  a  Main  Window  on  the  moititor  from  which  the  user  can: 

•  Load  and  display  a  stereo  image  from  a  pair  of  files. 

•  Load  and  display  a  stereo  image  from  a  database  of  cases. 

•  Control  various  functions  related  to  the  stereo  display  such  as  setting  the  brightness  and 
contrast,  inverting  grayscale,  inverting  depth,  invoking  highlight  mode,  and  changing  the 
magnification  factor. 

•  Invoke  Mouse  mode  in  either  of  two  initial  functions:  Brightness/Contrast  control  or 
Stereo  Cursor  control. 

The  GUI  Main  Window  is  used  primarily  for  loading  cases.  Although  the  GUI  provides 
controls  for  enhancing  the  stereo  display,  it  is  more  convenient  to  use  Mouse  Mode  for 
such  commands.  The  Main  Window  on  the  monitor  is  replaced  by  the  Mouse  Mode 
Window  when  the  application  is  in  Mouse  Mode.  Pressing  the  space  bar  will  bring  back 
the  GUI  Main  Window. 
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1.2  Mouse  Mode 


In  Mouse  Mode,  the  user  can  move  the  mouse  and  click  the  mouse  buttons  to  execute 
commands  that  control  two  separate  functions: 

•  Changing  the  brightness  and  contrast  of  the  display  (Brightness/Contrast  Mode) 

•  Moving  the  stereo  cursor  (Cursor  Mode). 

Mouse  Mode  is  invoked  automatically  in  Cursor  Mode  after  a  stereo  image  is  loaded.  It 
can  also  be  invoked  explicitly  from  the  Main  Window.  Pressing  the  space  bar  terminates 
Mouse  Mode  and  returns  to  the  GUI  Main  Window. 

Switching  between  Mouse  Functions 

The  user  can  toggle  between  the  two  types  of  Mouse  Mode  functions  by  pressing  the  F12 
key.  The  two  functions  are  mutually  exclusive  and  each  uses  the  mouse  movement  and 
mouse  buttons  for  different  purposes.  When  the  application  is  in  Mouse  Mode,  the  Mouse 
Mode  Window  replaces  the  Main  Window  on  the  monitor. 

When  the  Brightness/Contrast  Mode  is  active,  the  user  can  affect  the  stereo  display  as 
follows: 

•  Move  the  mouse  to  change  the  brightness  (horizontal  movement)  and  contrast  (vertical 
movement). 

•  Click  the  left  mouse  button  to  invert  the  grayscale  map. 

•  Click  the  right  mouse  button  to  invert  depth. 

•  Click  the  middle  button  to  temporarily  set  the  display  to  the  default  grayscale  values. 

In  Cursor  Mode,  the  user  can: 

•  Move  the  mouse  to  move  the  stereo  cursor  up,  down,  left  or  right. 

•  Hold  down  the  left  or  right  mouse  buttons  to  move  the  stereo  cursor  backward  or 
forward  respectively. 

•  Hold  down  the  noiddle  mouse  button  to  draw  a  straight  line. 

•  Hold  down  the  ENTER  key  on  the  keyboard  while  moving  the  mouse  to  draw  freehand 
lines. 


2.0  Stereo  Display  Windows 

The  Stereo  Display  application  be^ns  by  initializing  the  stereographies  hardware  md 
software.  Upon  completion  of  the  initialization,  the  user  sees  the  application’s  Main 
Window. 

2.1  The  GUI  Main  Window 

The  Main  Window  is  the  primary  GUI  window  for  the  application.  From  this  window  the 
user  can: 

•  Load  a  case  for  viewing  from  a  pair  of  files. 

•  Load  a  case  for  viewing  from  a  database. 

•  Get  information  about  the  case  currently  being  viewed. 

•  Save  a  new  case  to  the  database. 

•  Edit  an  existing  case  in  the  database. 

•  Execute  commands  that  affect  the  stereo  display. 


23 


Initially,  the  Main  Window  only  contains  a  Case  Selection  Frame  with  controls  for  selecting 
a  case  for  viewing.  Once  a  case  is  selected,  the  application  enters  Mouse  Mode  and  the 
Main  Window  is  replaced  by  the  Mouse  Mode  window.  When  the  user  exits  Mouse 
Mode,  the  Display  Controls  Frame  is  added  to  the  Main  Window.  In  the  Display  Controls 
frame  are  controls  for: 

•  Changing  the  brightness  and  contrast  of  the  stereo  image 

•  Inverting  the  grayscale 

•  Inverting  depdi 

•  Changing  the  magnification  factor 

•  Involang  Mouse  Mode 

The  Brightness/Contrast  controls  on  the  GUI  are  useful  in  that  they  allow  the  user  to  set  the 
grayscale  parameters  to  exact  values.  In  general,  however.  Mouse  Mode  is  much  more 
convenient  for  manipulating  the  stereo  display. 

The  Main  Window  also  provides  an  Auto-Select  Case  Frame  that  allows  the  user  to  move 
forward  or  backward  sequentially  through  the  case  database  to  select  cases  for  viewing. 

2.2  The  Mouse  Mode  Window 

The  Mouse  Mode  window  is  used  when  the  application  is  in  Mouse  Mode.  The  location  of 
the  mouse  pointer  in  the  window  controls  various  values  of  the  display  such  as  the 
brightness  or  contrast  settings,  the  X  and  Y  position  of  the  stereo  cursor,  and  the  location  of 
lines  drawn  in  the  display. 

When  in  Brighmess/Contrast  Mode,  a  small  square  is  drawn  into  the  Mouse  Mode 
Window.  The  square  is  located  at  the  default  brightness/contrast  settings  associated  with 
the  case.  The  user  can  quickly  view  the  stereo  image  using  the  default  settings  by  placing 
the  mouse  cursor  in  the  box. 

2.3  The  Registration  window 

This  window  contains  the  controls  for  aligning  the  images,  for  panning/scrolling,  and  for 
magnifying.  This  window  is  invoked  from  the  Tools->Registration  menu  on  the  Main 
Window. 


3.  Stereo  Display  Commands 

The  commands  provided  by  the  Stereo  Workstation  fall  into  four  general  categories: 

1.  Case  selection 

2.  Display  mode 

3.  Stereo  display  enhancements 

4.  Stereo  cursor  control 

The  following  sections  describe  the  commands  in  each  of  the  categories  and  how  they  are 
implemented  in  both  GUI  mode  and  in  Mouse  Mode. 
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3.1  Case  Selection 


A  case  is  comprised  of  two  x-ray  image  files:  the  left  eye  sees  one  image  file  and  the  right 
eye  sees  the  odier  image  file.  A  case  can  be  selected  for  viewing  in  two  ways: 

•  Explicitly  selecting  the  two  image  files  associated  with  the  case  from  disk 

•  Selecting  a  previously  sayed  case  from  a  database. 

3.1.1  Select  Case  Files  from  a  Disk 

GUI  Main  window: 

Load  Stereo  Pair  button 
File->Load  Stereo  Pair  menu 

This  command  allows  the  user  to  specify  the  two  image  files  for  viewing.  The  image  on  the 
stereo  screen  shows  the  data  as  they  are  in  the  raw  image  files,  without  any  enhancements. 

The  command  brings  up  a  custom  file  selection  dialog  that  allows  the  user  to  specify  the 
two  image  files  at  once.  The  first  file  selected  is  seen  by  the  left  eye  and  the  right  eye  sees 
the  second  file.  If  you  save  this  case  to  the  database,  the  full  pathnames  of  these  files  are 
saved  for  that  case.  Whenever  you  select  that  case  fi'om  the  database,  those  files  will  be 
reloaded. 

3.1.2  Select  a  Case  from  a  Database 

GUI  Main  window: 

Select  Case  from  Database...  button 

The  user  can  select  a  case  from  a  database  of  cases  viewed  previously  by  the  user  and  saved. 
When  the  user  selects  a  case  from  the  database,  the  system  loads  the  two  inaage  files  whose 
names  were  saved  with  the  case  and  initializes  the  display  to  the  default  settings  that  were 
saved  with  the  case.  On  the  Main  Window,  the  system  displays  the  case  number,-the  patient 
ID,  the  case  description,  and  the  names  of  the  image  files  that  the  user  is  viewing.  All  of  this 
information  was  obtained  from  the  database  when  the  case  was  selected. 

This  command  brings  up  the  "Select  A  Case"  dialog  window  with  the  database  of  cases 
shown  in  a  table.  The  user  selects  a  case  by  cMcking  on  the  row  that  describes  the  case. 
When  the  user  chcks  the  OK  button,  the  case  is  displayed. 

Note:  See  Section  3.1.4  Create  a  Database  for  how  to  create  a  database  file. 

3.1.3  Save  a  Case  to  the  Database 

GUI  Main  window: 

Save  Case...  button 
Case->Save  Case  menu 

The  system  allows  the  user  to  build  a  database  from  cases  that  were  read  from  disk  (see  the 
method  above),  or  to  change  the  information  for  a  case  that  was  read  from  the  database. 
After  loading  a  case,  either  from  disk  or  database,  the  user  can  optimize  the  image  by 
changing  grayscale  settings  and  registration.  When  the  display  is  optimized,  the  user  can 
then  save  the  case  under  a  unique  case  ID  number.  This  command  requires  the  user  to 
confirm  the  “save”  because  this  action  will  overwrite  any  data  previously  stored  for  this 
case.  Before  saving  the  case,  the  user  can  also  supply  a  patient  ID  and  case  description  by 
typing  directly  into  the  labeled  fields  in  the  Main  Window. 
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Saving  a  case  writes  the  following  information  about  that  case  to  the  database: 

•  A  unique  ID  number  assigned  by  the  system 

•  A  patient  ID  field  (text  from  the  Patient  data  entry  field  on  the  Main  Window) 

•  The  name  of  the  image  file  to  present  to  the  left  eye 

•  The  name  of  the  image  file  to  present  to  the  right  eye 

•  The  current  brightness  setting 

•  The  current  contrast  setting 

•  A  descriptive  text  field  (text  from  the  Description  data  entry  field  on  the  Main  Window) 

•  The  X  and  Y  offset  for  die  left  eye  image 

•  The  X  and  Y  offset  for  the  right  eye  image 

The  user  can  change  the  settings  or  descriptions  for  a  case  already  in  the  database  by  first 
selecting  the  case,  then  changing  settings  or  typing  in  the  Patient  or  Description  field  on  the 
Main  window,  and  finally  “saving”  the  case  again.  The  new  data  replaces  the  old  data  in 
the  database. 

This  command  brings  up  a  confirmation  dialog  that  asks  the  user  to  confirm  the  action. 

3.1.4  Create  a  Database 

When  you  save  a  case,  information  about  that  case  is  saved  to  a  database  file.  When  you 
load  a  case  from  the  database,  information  about  that  case  is  retrieved  from  the  database  file. 

The  following  steps  are  required  to  create  a  database  file. 

1.  Create  an  empty  file  with  any  name  (preferably  in  the  directory  containing  the  stereo 
pair  files.) 

2.  Identify  that  file  to  the  application  as  the  database  file  using  the  Case->Load  Case 
List...  menu  on  the  GUI  Main  Window. 

3 .  Use  the  Load  Stereo  Pair  command  to  load  a  stereo  pair. 

4.  Press  the  space  bar  to  exit  Mouse  Mode  and  return  to  the  GUI  Main  Window. 

5.  Click  the  Save  button  on  the  GUI  Main  Window  and  click  OK  to  confirm  the  "save". 

The  steps  above  will  write  out  the  first  case  to  the  database  file  in  the  correct  format. 
Subsequent  cases  that  you  save  will  be  added  to  this  file. 

You  can  use  the  Select  Case  from  Database...  button  to  see  the  database  table  created 
from  the  database  file. 

3.2  Display  Mode 

GUI  Main  window: 

Display->Stereo  menu 
Display->Mono  menu 

The  user  can  choose  a  stereo  or  a  monaural  view  of  a  case.  In  the  stereo  display,  the 
perception  of  a  three-dimensional  image  is  achieved  by  presenting  one  image  to  the  left  eye, 
while  presenting  a  different  image,  acquired  from  a  slightly  different  point-of-view,  to  the 
right  eye.  The  human  visual  system  is  able  to  fuse  these  two  images  into  one  containing 
depth. 

A  mono  display  simply  presents  the  same  image  (in  this  case  the  image  presented  to  the  left 
eye)  to  both  eyes.  The  perceived  image  has  no  depth. 
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3.3  Display  Enhancements 

Before  they  are  displayed  on  the  screen,  the  8-bit  data  values  in  an  image  file  pass  through  a 
look-up  table  that  maps  each  of  the  input  data  values  into  an  output  display  value  between  0 
(full  white)  and  255  (full  black).  By  default,  the  table  maps  each  input  data  value  onto  itself. 
The  resulting  picture  on  the  display  is  a  true  representation  of  the  image  data  as  it  exists  in 
the  file.  The  user  can  change  the  mapping  of  input-to-output  values  to  enhance  the 
appearance  of  the  picture  on  the  screen.  By  changing  the  mapping  the  user  can  change 
visual  aspects  of  the  display  such  as  brightness  and  contrast,  create  a  “negative”  of  the 
image,  or  highlight  a  specific  data  value. 

Grayscale  manipulation  can  be  done  in  two  ways: 

1 .  By  using  the  Brightness/Contrast  controls  on  the  GUI  Main  Window 

2.  By  the  mouse  in  Brightness/Contrast  Mode. 

3.3.1  Increase/Decrease  Brightness 
GUI  Main  Window: 

Scrollbar  attached  to  the  “Brightness”  rectangle 
Rectangle  labeled  “Brightness” 

Mouse  Mode  (Brightness/Contrast  Model: 

Move  the  mouse  left  or  right. 

This  command  increases  or  decreases  the  overall  brightness  level  of  the  image  from  its 
current  setting.  The  value  of  the  brightness  level  ranges  from  0  to  255.  The  user  can 
change  this  value  in  large  or  small  increments  as  needed.  For  the  default  linear  map,  the 
higher  the  value  the  brighter  the  display  (if  the  map  is  inverted,  the  higher  the  value  the 
darker  the  display). 

The  controls  on  the  GUI  Main  Window  allow  the  user  to  make  discrete  changes  to  the 
brightness  value.  With  the  scrollbar,  the  user  can  make  small  changes  to  the  value  by 
clicking  the  left  or  right  arrows,  or  make  larger  changes  by  dragging  the  slider  or  clicking 
arrows.  With  the  Brightness  rectangle,  the  user  can  increase  or  decrease  the  value 
continuously  by  placing  the  cursor  in  the  rectangle  then  holding  down  either  the  right  or  left 
mouse  button  respectively.  Releasing  the  mouse  button  stops  die  change. 

Mouse  Mode  for  controlling  brightness  and  contrast  is  invoked  by  clicking  the 
Brightness/Contrast  button  in  the  “Mouse  Controllers”  area  of  the  GUI  Main  Window. 
When  the  Mouse  Mode  window  appears,  the  user  can  move  the  mouse  to  the  light/left  to 
increase/decrease  brightness. 

Note:  In  Mouse  Mode,  the  ability  to  change  brightness  is  available  only  from 
Brightness/Contrast  Mode.  If  the  title  bar  of  the  Mouse  Mode  Window  reads  "Cursor 
Mode",  then  press  the  F12  key  on  the  keyboard  to  switch  to  Brightness/Contrast  Mode. 

To  exit  this  window  and  preserve  the  current  settings,  press  the  space  bar.  The  current  value 
for  brightness  will  register  on  the  brightness  controls  on  the  Main  window. 
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3.3.2  Increase/Decrease  Contrast 
GUI  Main  Window: 

Scrollbar  attached  to  the  “Contrast”  rectangle 
Rectangle  labeled  “Contrast” 

Mouse  Mode  rBriehtness/Contrast  Mode): 

Move  the  mouse  up  or  down. 

This  command  increases  or  decreases  the  range-of-contrast  value.  The  conU’ast  range  value 
is  a  number  between  0  and  255  that  defines  the  width  of  a  window  about  the  brightness 
level.  Data  values  in  this  window  will  be  mapped  from  white  (0)  to  black  (255).  AU  data 
values  below  this  range  will  be  mapped  to  white  and  all  data  values  above  this  range  will  be 
mapped  to  black.  Increasing  the  contrast  range  decreases  the  amount  of  contrast  in  the 
display:  conversely,  decreasing  the  contrast  range  increases  the  amount  of  contrast  in  the 
image. 

These  commands  on  the  GUI  Main  Window  are  implemented  in  the  same  manner  as  the 
Brightness  level  controls  and  commands.  (See  Increase/Decrease  Brightness  section 
presented  earlier  more  information.) 

Mouse  Mode  to  control  brightness  and  contrast  is  invoked  by  clicking  the 
Brightness/Contrast  button  in  the  “Mouse  Controllers”  area  of  the  Main  Window.  When 
the  Mouse  Mode  window  appears,  the  user  can  move  the  mouse  up/down  to 
increase/decrease  contrast. 

When  a  case  is  first  displayed,  the  brightness  and  contrast  values  are  set  to  default  values 
(either  to  256  and  128  respectively,  or  to  values  that  were  stored  with  the  case  in  the 
database).  However,  once  the  mouse  is  moved,  however  slightly,  the  system  computes  new 
brightness  and  contrast  values  based  on  the  position  of  the  mouse  cursor  in  the  Mouse 
Mode  Window.  You  can  temporarily  restore  the  default  values  by  either  holding  down  the 
middle  mouse  button  or  by  pressing  the  Shift  key. 

To  exit  this  window  and  preserve  the  current  settings,  press  the  space  bar.  The  current  value 
for  contrast  will  register  on  the  contrast  controls  on  the  Main  window. 

Note;  In  Mouse  Mode,  the  ability  to  change  contrast  is  available  only  from 
Brightness/Contrast  Mode.  If  the  title  bar  of  the  Mouse  Mode  Window  reads  "Cursor 
Mode",  then  press  the  F12  key  on  the  keyboard  to  switch  to  Brightness/Contrast  Mode. 

3.3.3  Reset  Grayscale 

GUI  Main  Window: 

Reset  button. 

Mouse  Mode  (Brightness/Contrast  Mode): 

Press  and  hold  down  the  Middle  Mouse  button 

This  command  resets  the  brightness  level  and  contrast  range  to  their  default  values  of  128 
and  256  respectively.  If  the  case  being  viewed  was  read  from  the  database,  then  the  levels 
will  be  reset  to  the  values  stored  with  the  case. 

The  Mouse  Mode  command  temporarily  restores  the  default  values  so  that  you  can  quickly 
view  the  image  with  the  default  settings.  When  you  release  the  mouse  button  or  move  the 
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mouse,  new  values  will  be  computed  based  on  the  position  of  the  mouse  pointer  in  the 
Mouse  Mode  Window. 

Note:  In  Mouse  Mode,  the  ability  to  reset  the  grayscale  is  available  only  from 
Brightness/Contrast  Mode.  If  the  title  bar  of  the  Mouse  Mode  Window  reads  "Cursor 
Mode",  then  press  the  F12  key  on  the  keyboard  to  switch  to  Brightness/Contrast  Mode. 

3.3.4  Highlight  a  Data  Value 

GUI  Main  Window: 

Highlight  check  box 

Mouse  Mode : 

Not  implemented. 

This  command  initially  sets  the  contrast  range  to  one  and  then  displays  the  input  data  value 
that  maps  to  the  brightness  level  as  pure  white,  thus  highlighting  that  one  data  value.  The 
remainder  of  the  data  is  mapped  linearly  into  values  between  0  and  126.  This  allows  the 
highlighted  value  to  be  observed  against  a  dimmed  version  of  the  image. 

When  highlighting  is  enabled,  the  image  is  set  to  half  of  its  full  brightness  and  only  the  data 
whose  v5ue  equals  the  current  brightness  level  is  mapped  into  pure  white.  The  user  can 
highlight  single  data  values  by  leaving  the  contrast  range  at  one  and  changing  the  brightness 
level,  or  highlight  a  range  of  values  about  the  brightness  level  by  changing  the  contrast 
range.  Turning  highli^ting  off  returns  the  image  to  the  visual  state  prior  to  the  “highlight 
on”  command. 

3.3.5  Invert  Grayscale 

GUI  Main  Window: 

Invert  GRAYSCALE  button. 

Mouse  Mode  CBrightness/Contrast  Model: 

Click  the  left  button. 

This  toggle  command  inverts  the  linear  ^ayscale  mapping.  Normally,  input  data  are 
mapped  linearly  into  output  values  ranging  from  pure  white  (0)  to  pure  black  (255). 
Inverting  the  map  causes  the  output  values  for  the  same  input  values  to  range  from  black-to- 
white  instead.  This  has  the  effect  of  producing  a  “negative”  of  the  image.  Inverting  the 
grayscale  again  returns  the  mapping  and  display  to  its  former  state. 

Note:  In  Mouse  Mode,  the  ability  to  invert  grayscale  is  available  only  from 
Brightness/Contrast  Mode.  If  the  title  bar  of  the  Mouse  Mode  Window  reads  "Cursor 
Mode",  then  press  the  F12  key  on  the  keyboard  to  switch  to  Brightness/Contrast  Mode. 

3.3.6  Save  Grayscale  Settings 

The  “Save  Case”  command  described  earlier  allows  the  user  to  save  the  current  brightness 
and  contrast  settings  for  a  case  so  that  when  the  case  is  recalled  at  a  later  time,  it  will  appear 
as  it  did  when  it  was  saved.  (See  Section  3.1.3  Save  a  Case  to  the  Database.) 
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3.3.7  Invert  Depth 

GUI  Main  window: 

Invert  DEPTH  button 

Mouse  Mode  (Brightness/Contrast  Mode): 

Click  the  right  button. 

This  toggle  command  switches  the  image  presentation  to  each  eye  (i.e.,  the  image  presented 
to  the  left  eye  is  shown  to  the  right  eye  instead  and  vice  versa).  The  effect  is  an  inversion  in 
depth;  whatever  appears  to  have  been  at  the  back  of  the  display  now  appears  in  front  and 
what  was  in  front  appears  to  have  moved  to  the  back. 

Note:  In  Mouse  Mode,  the  ability  to  invert  depth  is  available  only  from  Brightness/Contrast 
Mode.  If  the  title  bar  of  the  Mouse  Mode  Window  reads  "Cursor  Mode",  then  press  the 
F12  key  on  the  keyboard  to  switch  to  Brightness/Contrast  Mode. 

3.3.8  Adjust  Alignment 

This  command  is  invoked  from  the  GUI  Registration  Window.  To  see  this  window,  select 
the  Tool->Registration  menu  on  the  GUI  Main  Window. 

GUI  Registration  Window: 

The  alignment  controls  are 

Scrollbar  attached  to  a  rectangle  identified  by  eye  and  direction 
Rectangle  identified  by  eye  and  direction 

This  capability  allows  the  user  to  move  each  of  the  images  up,  down,  left  or  right 
independently  to  correct  for  any  misalignment  of  the  images  that  may  have  occurred  when 
the  pictures  were  taken. 

Each  combination  of  eye  and  direction  (left/horizontal,  left/vertical,  right/horizontal,  and 
right/vertical)  has  two  controls,  similar  to  the  controls  used  for  brightness  and  contrast. 

With  these  controls,  the  user  can  move  each  image  independently  in  the  horizontal  (left  and 
right)  or  vertical  (up  and  down)  direction. 

To  operate  any  of  the  four  scrollbars,  the  user  clicks  the  left  or  right  arrows  to  move  the 
image  by  small  increments,  or  drags  the  slide  or  clicks  in  the  track  to  move  in  larger 
increments.  To  use  any  of  the  eye/direction  rectangles,  the  user  places  the  cursor  in  the 
rectangle  then  holds  down  either  the  left  or  right  mouse  button  to  move  the  image 
continuously  in  one  direction  (left  or  up)  or  the  opposite  direction  (right  or  down) 
respectively.  Releasing  the  mouse  button  stops  the  movement. 

The  user  can  save  these  settings  by  clicking  the  OK/Save  button  when  exiting  the  window. 
If  the  user  clicks  Cancel  to  exit  the  window,  the  display  is  reset  to  the  settings  in  effect  prior 
to  invoking  the  Registration  window. 

3.3.9  Pan  or  Scroll  Image 

This  command  is  invoked  from  the  GUI  Registration  Window.  To  see  this  window,  select 
the  Tool->Registration  menu  on  the  GUI  Main  Window. 

GUI  Registration  window: 

Pan/Scroll  check  box 
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Mouse  Mode  rSither  Mode): 

Scrolling:  Press  the  Up  or  Down  arrows  on  the  keyboard  (not  the  keypad). 
Panning:  Not  implemented. 

This  command  allows  the  user  to  move  the  image  on  the  display  up  or  down  (scrolling),  or 
left  or  right  (panning).  In  this  case  the  left  and  right  eye  images  move  together. 

To  enable  panning  or  scrolling  from  the  GUI  Registration  Window,  the  user  clicks  the 
pan/scroll  check  box  to  place  an  “X”  in  the  box.  To  disable  movement,  the  user  clicks  in 
the  check  box  again  to  remove  the  “X”.  When  panning  and  scrolling  are  enabled,  the 
images  are  coupled  together  and  the  alignment  controls  for  the  left  eye  serve  to  move  the 
image  on  the  display  up,  down,  left,  or  right.  The  alignment  controls  for  the  right  eye  are 
disabled.  When  panning  and  scrolling  are  disabled,  the  images  are  uncoupled,  the 
alignment  controls  for  the  right  eye  are  enabled,  and  the  user  can  again  move  each  image 
independently. 

In  Mouse  Mode,  the  images  are  always  coupled. 

3.3.10  Zoom  Image  IX,  2X,  or  4X 

GUI  Main  window: 

IX,  2X,  and  4X  radio  buttons 

Mouse  Mode: 

Not  implemented. 

The  user  can  enlarge  the  image  on  the  display  by  a  factor  of  1, 2  or  4.  The  image  is 
magnified  about  the  center  of  the  current  view. 

The  user  selects  a  magnification  factor  by  clicking  one  of  the  radio  buttons. 

3.4  Stereo  Cursor  Control 

Sometimes  the  user  may  want  to  point  to  some  interesting  feature  in  the  image.  The 
application  provides  a  large,  crosshair  cursor  on  the  stereo  display  that  the  user  can  position 
horizontally,  vertically,  or  in  depth  with  the  aid  of  the  mouse.  The  mouse  is  able  to  control 
the  cursor  when  the  application  is  in  Mouse  Mode  for  controlling  the  cursor  (Cursor 
Mode). 

Note:  In  Mouse  Mode,  commands  that  control  the  stereo  cursor  are  available  only  from 
Cursor  Mode.  If  the  title  bar  of  the  Mouse  Mode  Window  reads  "Brightness/Contrast 
Mode",  then  press  the  F12  key  on  the  keyboard  to  switch  to  Cursor  Mode. 


3.4.1  Display  a  Crosshair  Cursor 

After  loading  an  image,  the  system  enters  Mouse  Mode  in  Cursor  Mode  and  the  user  will 
see  a  large  crosshair  cursor  drawn  into  the  stereo  image.  Cursor  Mode  can  also  be  invoked 
by  clicking  the  Stereo  Cursor/Line  Drawing  button  in  the  “Mouse  Controllers”  area  of  the 
Main  Window.  When  the  Mouse  Mode  window  appears,  the  user  can  use  the  mouse  to 
manipulate  the  cursor  as  described  in  the  sections  that  follow. 
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3.4.2  Move  the  Crosshair  Cursor  within  a  Depth  Plane 

Mouse  Mode  (Cursor  Mode): 

Move  the  mouse  left,  right,  up  or  down. 

The  user  can  move  the  crosshair  cursor  left,  right,  up,  or  down  by  moving  the  mouse.  This 
X/Y  movement  takes  place  in  whatever  depth  plane  the  cursor  is. 

3.4.3  Move  the  Crosshair  Cursor  to  Different  Depth  Planes 

Mouse  Mode  (Cursor  Model: 

Press  and  hold  the  left  mouse  button  to  move  backward. 

Press  and  hold  the  right  mouse  button  to  move  forward. 

In  addition  to  moving  horizontally  and  vertically,  the  crosshair  cursor  can  also  move  forward 
and  backward  in  depth,  giving  the  appearance  of  moving  in  and  out  of  the  image  itself. 

Once  the  user  positions  the  cursor  in  depth,  then  moving  the  mouse  moves  the  cursor 
horizontally  and  vertically  in  that  depth  plane. 

The  cursor  will  continue  to  move  in  depth  as  long  as  the  left  or  right  mouse  button  is  held 
down.  Releasing  the  button  stops  the  movement  in  depth.  The  cursor  remains  at  that 
depth  plane  until  another  command  to  "move  in  depth"  is  executed.  Any  horizontal  or 
vertic^  movement  of  the  mouse  now  moves  the  cursor  around  in  that  depth  plane. 

3.4.4  Drawing  using  the  Crosshair  Cursor 


Mouse  Mode:  (Cursor  Model 
Draw  a  straight  line  in  any  depth  plane: 

1 .  If  a  different  depth  plane  is  desired,  press  the  mouse  buttons  to  move  the  cursor  to  the 
desired  plane  then  release  the  mouse  buttons. 

2.  To  begin  the  line,  hold  the  middle  mouse  button  down  while  moving  the  cursor  around 
the  screen. 

3.  To  end  the  line,  release  the  mouse  button. 

The  user  can  use  the  Crosshair  Cursor  to  draw  straight  lines  in  a  plane  or  in  depth.  The 
middle  mouse  button  enables  “LineDraw”  mode.  To  draw  a  straight  hne,  first  move  the 
cursor  to  the  place  where  the  line  will  begin,  then  press  and  hold  the  middle  mouse  button. 
Now  when  you  move  the  cursor  horizontSly,  vertically,  or  in  depth,  a  line  will  be  drawn 
from  the  beginning  point  to  the  current  location  of  the  cursor,  giving  the  appearance  of  a 
“rubber  band”  line.  When  you  reach  the  position  at  which  you  want  the  line  to  end,  release 
the  mouse  button. 

Mouse  Mode:  (Cursor  Model 
Delete  the  last  straight  line  drawn'. 

Press  the  DELETE  KEY. 

Redraw  the  last  straight  line  deleted'. 

Press  the  DELETE  KEY  again. 

You  can  delete  the  line  last  drawn,  by  pressing  the  DELETE  KEY.  You  can  bring  it  back  by 
pressing  the  DELETE  KEY  again. 
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Mouse  Mode:  (Cursor  Mode) 

Draw  a  freehand  line: 

Hold  the  ENTER  KEY  down  while  moving  the  cursor. 

You  can  also  do  freehand  drawing.  The  ENTER  KEY  enables  freehand  drawing.  To  draw 
freehand,  hold  down  the  ENTER  KEY  while  moving  the  cursor  around  the  screen.  You 
cannot  delete  a  freehand  line. 

Mouse  Mode:  (Cursor  Model 
Clear  the  screen  of  all  lines: 

Press  the  CTRL  and  DELETE  KEYS. 

To  clear  the  screen  of  all  lines,  hold  the  CTRL  KEY  down  then  press  the  DELETE  KEY. 
This  will  actually  reload  a  new  image. 

Mouse  Mode:  (Cursor  Mode') 

Draw  a  straight  line  backward  or  forward  in  depth: 

Hold  the  middle  mouse  button  down  while  simultaneously  pressing  the  left  or  right 
mouse  button  to  draw  a  line  backward  or  forward  in  depth.  To  draw  a  line  diagonally 
in  depth,  drag  the  mouse  while  holding  the  mouse  buttons  down. 
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APPENDIX  B:  Slides  from  Talk  at  Far  West  ’99  Image  Perception  Conference 
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Stereoscopic  Digital  Mammography: 
Improving  Diagnostic  Accuracy 
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Goals  of  the  Project 

■  Acquire  stereo  mammograms  from  about  250  women  patients 
scheduled  for  biopsy  of  a  focal  breast  lesion. 

■  Improve  diagnostic  accuracy  using  stereo  mammography. 

■  Develop  understanding  of  new  stereo-based  visual  features. 

■  Using  ROC  methods,  compare  the  diagnostic  accuracy  of 
readings  attained  with: 

♦  1.  Standard  films  alone 

♦  2.  Standard  films  +  non-stereo  viewing  of  digital  images 

♦  3.  Standard  films  +  stereo  viewing  of  digital  images 


Why  Stereo  Mammography? 

■  Digital  mammography  devices  and  better  stereo  display 
methods  now  make  stereo  mammography  feasible. 

■  Potential  to  improve  detection  of  early,  subtle  breast 
lesions. 

■  Potential  to  improve  the  ability  to  distinguish  malignant 
from  benign  lesions. 
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A  Significant  Problem  with  Standard 
Mammography 


■  Difficult  to  integrate  information  from  a  pair  of 
orthogonal  2-D  views,  to  mentally  construct  the  3-D 
structure  of  the  breast. 

■  In  stereo,  one  can  directly  see  the  internal 
3-D  structure. 


Gains  in  Diagnostic  Accuracy 
from  Stereo 

■  Calcifications 

♦  One  can  directly  see  the  3-D  shapes  of  individual  elements  of 
calcium. 

♦  One  can  directly  see  the  3-D  geometric  distribution  of  a  cluster  of 
calcifications. 

■  Masses  and  Architectural  Distortions 

♦  Easier  to  see  a  mass  or  architectural  distortion  because  of  the 
separation  in  depth  of  nearby  overlying  and  underlying  glandular 
tissue. 

♦  Easier  to  judge  characteristics  of  the  surface  or  border  of  a  mass,  or 
the  structure  of  an  architectural  distortion,  because  it  is  seen  directly 
as  a  volumetric  shape. 
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Placeholder:  GE  digital  mammography 

device 


Stereo  Image  Acquisition 


x-ray 


\  / 
\  / 

\  / 

\  / 
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Placeholder:  Stereo  display  system 


Stereo  Mammogram  Display  System 

■  Prototype  system  based  on  a  PC  with  a  Matrox  image  display 
controller  with  dual  image  buffers  and  stereo  circuitry. 

■  Stereo  pair  of  images  viewed  through  CrystalEyes  LCD 
stereo  eyeglasses  on  Hitachi  21”  color  monitor  capable  of 
synching  to  120  Hz  vertical  refresh  rate. 

■  BBN-developed  software  for  selecting  and  manipulating 
appearance  of  the  displayed  images,  with  control  over: 

♦  Brightness  and  contrast  windowing  of  images 

♦  Inversion  of  grayscale 

♦  Inversion  of  depth 

♦  Scrolling  and  magnification  level 
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Placeholder:  Stereo  mammogram  -  Left 


Placeholder:  Stereo  mammogram  -  Right 
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Preliminary  Reading  Study 

■  Case  set:  76  cases 


Lesion  type 

Truth 

Benign  Malignant 

Total 

Mass 

22 

12 

34 

Calcifications 

29 

4 

33 

Architectural 

Distortion 

2 

7 

9 

Total 

53 

23 

76 

■  4  highly-experienced  mammographers  as  readers 

■  Each  reader  read  each  case  in  3  stages,  judging  the 
probability  of  malignancy  at  each  stage: 

♦  Stage  1:  Films  alone  (using  films  from  study  leading  to  biopsy) 

♦  Stage  2:  Films  +  non-stereo  digital  images 

♦  Stage  3:  Films  +  stereo  digital  images 

■  Each  reader  was  asked  to  log  noteworthy  comments  about 
individual  cases 

■  Accuracy  of  the  judgment  data  was  analyzed  using  ROC 
methods 
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True  Positive  Probability 


ROC’s  of  Film  Alone  vs. 
Film  +  Non-stereo  Digital  Images 


False  Positive  Probability 


ROC’s  of  Film  Alone  vs. 
Film  +  Stereo  Digital  Images 


False  Positive  Probability 


Improved  Accuracy  by  Stereo  Limited  by 
Performance  on  Calcification  Cases 

■  The  net  improvement  in  accuracy  through  use  of  stereo 
mammography  is  underestimated  in  these  data  because  of  the 
poor  visibility  of  calcifications  in  the  digital  display. 

■  All  4  readers  logged  comments  that  many  small,  fine 
calcifications,  which  were  visible  in  the  film,  were  not 
visible  on  the  digital  images— affecting  at  least  13  of  the  33 
calcification  cases. 

■  Poor  visibility  of  small  calcifications  due  to  characteristics  of 
the  display  monitor  (color  rather  than  grayscale),  and 
possibly  due  also  to  the  100  micron  spot  size  of  the  GE 
mammography  unit. 


Improvement  in  Prediction  Probability  by 
Type  of  Lesion 


lnf)rov«m»nt  In  Prcdictbn  Probability 
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Stereo  Helps  in  Characterizing  Masses  and 
Architectural  Distortion 

■  By  separating  obscuring  parenchymal  tissue  from  a  mass  in 
depth,  stereo  helps  define  the  border  characteristics  and  shape 
of  a  mass  (7  cases). 

■  By  localizing  a  density  in  the  volume,  stereo  helps  one  see 
that  an  apparent,  but  questionable,  density  or  distortion  on 
film  either  is,  or  is  not,  real. 

♦  Apparent  architectural  distortion  on  film  disappears  in  stereo,  with 
large  (correct)  change  in  predicted  probability  for  3  of  4  readers  (1 
benign  case). 

♦  Architectural  distortion  seen  more  confidently  in  stereo  (1  malignant 
case). 


Stereo  Helps  in  the  Detection  of  Masses  and 
Architectural  Distortion  Not  Seen  On  Film 

■  Significantly,  in  6  of  the  76  cases,  a  mass  or  architectural 
distortion  was  detected  in  the  stereo  images  which  was  not 
visible  on  the  films. 

♦  In  3  cases,  a  mass  was  detected  in  the  stereo  image  which  was  not 
visible  on  the  film. 

♦  In  3  cases,  a  region  of  architectural  distortion  was  detected  in  the 
stereo  image  which  was  not  visible  on  the  film. 

■  The  basis  for  this  increased  detection  sensitivity  is  most 
likely  the  additional  information  conveyed  by  the  volumetric 
display  of  breast  tissue,  permitting  more  sensitive  direct 
detection  of  a  volumetric  focal  density  or  of  spherically 
radiating  strands  of  tissue. 
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Next  Steps 

■  Continue  accruing  new  stereo  image  cases  to  enlarge  our 
sample  for  a  final  reading  study. 

■  Acquire  better  stereo  display  monitor: 

♦  Grayscale,  21” 

♦  Capable  of  synching  to  120  Hz  vertical  refresh 

♦  Large  range  of  vertical  size  adjustment  (tol/2  screen  height) 

♦  Fast  decaying  phosphor  (to  minimize  ghosting  between  eyes) 

■  Consider  using  stereo  film  display  as  well  as  digital  display. 

■  Try  to  improve  calcification  classification  by  exploiting 
unrecognized  diagnostic  information  carried  by  the 
volumetric  geometry  of  the  calcium  elements. 


Demonstration  of  Stereoscopic  Digital 
Mammography 

■  Stereo  mammograms  of  several  cases,  illustrating  stereo 
views  of  masses,  architectural  distortion,  and  micro- 
calcifications. 

■  Images  presented  in  hand-held  stereo  viewers,  each 
containing  a  stereo  pair  of  35mm  slides  of  one  case. 
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